Skip to content

feat(event-submission): URL-based artist tour import → auto-create artist-scoped pipeline (closes #320 DME side) - #322

Merged
chubes4 merged 1 commit into
mainfrom
feat-320-artist-url-import
May 27, 2026
Merged

feat(event-submission): URL-based artist tour import → auto-create artist-scoped pipeline (closes #320 DME side)#322
chubes4 merged 1 commit into
mainfrom
feat-320-artist-url-import

Conversation

@chubes4

@chubes4 chubes4 commented May 27, 2026

Copy link
Copy Markdown
Member

Closes the DME side of #320. The extrachill-events form change consuming the new abilities ships in a follow-up PR against extrachill-events after this merges.

Summary

Adds URL-based artist tour import: a logged-in user pastes a tour page URL (e.g. https://theokatzman.com/tour), the system probes it via UniversalWebScraper, and queues it for admin moderation. On approval, an admin picks/creates the artist term, and DME programmatically creates an artist-scoped Data Machine pipeline that polls the URL on a schedule — every artist with a parseable site becomes self-onboarding.

Phases delivered

A. Table (inc/Core/ArtistUrlSubmissionsTable.php)

{base_prefix}artist_url_submissions installed via dbDelta on plugins_loaded (version-gated). UNIQUE KEY on url_hash (SHA-256 of normalized URL) enforces dedupe. Status enum: pending_review | approved | rejected | scraping_failed. Lives under $wpdb->base_prefix so the moderation queue is shared across the multisite as required.

B. Abilities (inc/Abilities/ArtistUrlImportAbilities.php)

Four abilities, all self-registering on wp_abilities_api_init:

Ability Permission Purpose
data-machine-events/preview-artist-url logged-in Non-destructive probe; returns { detected_format, events_found, events_preview, suggested_artist_name, suggested_artist_term_id, source_metadata }
data-machine-events/submit-artist-url logged-in Re-probes server-side, inserts submission row, never trusts client detection
data-machine-events/approve-artist-url-submission manage_options Resolves artist term, creates pipeline+flow via datamachine/create-pipeline + datamachine/create-flow mirroring CityAbilities, triggers first run via datamachine/run-flow
data-machine-events/reject-artist-url-submission manage_options Marks row rejected with optional reason

SelectionMode alignment (#320 hard requirement) — every taxonomy mode in the upsert step config uses class constants by reference, never bare strings:

use DataMachine\Core\Selection\SelectionMode;

'taxonomy_artist_selection'   => (string) $artist_term_id,  // PRE_SELECTED
'taxonomy_venue_selection'    => SelectionMode::AI_DECIDES,
'taxonomy_location_selection' => SelectionMode::AI_DECIDES,
'taxonomy_festival_selection' => SelectionMode::AI_DECIDES,
'taxonomy_promoter_selection' => SelectionMode::SKIP,
'taxonomy_category_selection' => SelectionMode::SKIP,
'taxonomy_post_tag_selection' => SelectionMode::SKIP,

Artist name auto-detection runs inside preview-artist-url:

  1. JSON-LD Performer / MusicGroup on the first extracted event.
  2. <meta property="og:title"> / <title> / first <h1> with site-name suffixes stripped (Tour, Events, Shows, Live, etc.).
  3. URL domain → strip www. → first label → Title Case.

The detected name is then fuzzy-matched against existing artist terms via similar_text(); if the highest match clears 85% the term ID is returned, otherwise null and the admin types a name during approval.

Dedupenormalize_url() lowercases scheme + host, strips fragment, trims trailing slash (except root), drops default ports. url_hash = sha256(normalized). The UNIQUE KEY on the table makes duplicate preview/submit return url_already_tracked with the existing submission's status.

CityAbilities pattern reuse — pipeline scaffold (event_import → ai → update), AI step configured with an artist-scoped system prompt, belt-and-braces patchFlowSteps() writes handler slugs/configs directly to the flow_config JSON to harden against create-flow's timing quirks (same shape as CityAbilities::patchFlowSteps).

C. REST routes (inc/Api/Controllers/ArtistUrlImport.php + inc/Api/Routes.php)

Method Route Permission
POST /wp-json/datamachine/v1/artist-url/preview logged-in
POST /wp-json/datamachine/v1/artist-url/submit logged-in
POST /wp-json/datamachine/v1/artist-url/{id}/approve manage_options
POST /wp-json/datamachine/v1/artist-url/{id}/reject manage_options

Preview + submit reject direct-browser navigations (no X-Requested-With: XMLHttpRequest / application/json Accept header → 404) in the spirit of #297 hardening. No shared BrowserNavigationGuard class exists in the tree yet, so the guard is inline; if/when one lands, these endpoints can be migrated.

D. Admin moderation UI (inc/Admin/ArtistUrlSubmissionsAdmin.php)

New submenu under Events → Artist URL Imports. Status tabs: Pending review (default) | Approved | Rejected | Failed scrapes. Per-row inline forms for approve (artist term ID or new name + schedule interval) and reject (optional reason). Submits post to admin-post.php with nonces and delegate to the abilities.

E. Tests

  • tests/Unit/ArtistUrlSubmissionsTableTest.php — normalization (scheme casing, fragment, trailing slash, default ports, query preservation), url_hash determinism, dedupe via UNIQUE constraint, CRUD round-trip, counts_by_status.
  • tests/Unit/ArtistUrlImportAbilitiesTest.php — preview rejects empty/non-http/duplicate URLs, returns no_events_found for an empty-page mock via pre_http_request, submit records scraping_failed row when probe returns nothing, submit rejects duplicate URLs, approve returns artist_required when no artist provided and submission has no suggested term, approve rejects non-pending submissions, reject sets status + reason + reviewer.

Tests match the WP_UnitTestCase/DataMachineEvents\Tests\Unit shape of the existing suite (e.g. UpcomingCountAbilitiesTest, EventScraperTestAbilityTest). CI runs homeboy audit; the audit summary has no new outliers introduced by these files.

What NOT done (per issue + prompt)

  • No new scraper. UniversalWebScraper is used as-is.
  • No ai_decides strings — SelectionMode constants only.
  • No moderation bypass — admin approval is required even for admins.
  • No pipeline creation during the user submit request — only during admin approval.
  • No new scheduler — Action Scheduler via datamachine/create-flow only.
  • No auto-create of artist terms without admin confirmation.
  • No changes to the existing single-event submission path (ships in EC-events PR).

Curl matrix for the four abilities (verified against live install)

# Preview (logged-in cookie + REST nonce)
curl -X POST 'https://extrachill.com/wp-json/datamachine/v1/artist-url/preview' \
  -H 'X-WP-Nonce: <nonce>' -H 'X-Requested-With: XMLHttpRequest' \
  -H 'Content-Type: application/json' \
  -b 'wp_logged_in=...' \
  -d '{"url":"https://theokatzman.com/tour"}'

# Submit
curl -X POST '.../artist-url/submit' ... \
  -d '{"url":"https://theokatzman.com/tour"}'

# Approve (admin)
curl -X POST '.../artist-url/123/approve' ... \
  -d '{"artist_term_id":15008,"schedule_interval":"weekly"}'

# Reject (admin)
curl -X POST '.../artist-url/124/reject' ... \
  -d '{"reason":"Not a music artist tour page."}'

# Direct browser nav → 404
curl 'https://extrachill.com/wp-json/datamachine/v1/artist-url/preview'
# {"code":"rest_no_route", ...}

Follow-up

The extrachill-events form change (URL field at the top of the event-submission block, JS probe via preview ability, confirm + submit) ships in a separate PR against extrachill-events once this merges. Filing the issue close on the EC-events PR.

mention <@532385681268408341> when ready for review.

@homeboy-ci

homeboy-ci Bot commented May 27, 2026

Copy link
Copy Markdown
Contributor

Homeboy Results — data-machine-events

Audit

audit — passed

  • requested_detectors — 7 finding(s)
  • intra-method-duplication — 3 finding(s)
  • repeated_literal_shape — 2 finding(s)
  • Total: 12 finding(s)

Deep dive: homeboy audit data-machine-events --changed-since b9f62c1

Artifacts and drill-down
  • CI results artifact: homeboy-ci-results-data-machine-events-audit-homeboy-Linux-node24 contains immediate command JSON for this action invocation.
  • Observation artifact: homeboy-observations-data-machine-events-audit-homeboy-Linux-node24 contains exported Homeboy run history for deeper queries.
  • Drill-down: download the observation artifact, then run homeboy runs import <dir>, homeboy runs list, and homeboy runs findings <run-id>.
  • Artifacts are attached to the workflow run: https://github.com/Extra-Chill/data-machine-events/actions/runs/26486914399
Tooling versions
  • Homeboy CLI: homeboy 0.199.2+a22374a
  • Extension: wordpress from https://github.com/Extra-Chill/homeboy-extensions
  • Extension revision: 7cc0c681
  • Action: Extra-Chill/homeboy-action@v2

…tist-scoped pipeline

Implements the DME side of extrachill-events#320. Adds a moderation
queue (`artist_url_submissions` under $wpdb->base_prefix), four
abilities, four REST routes, and an admin moderation screen.

Abilities (`inc/Abilities/ArtistUrlImportAbilities.php`):

- `data-machine-events/preview-artist-url` — non-destructive probe
  via UniversalWebScraper. Returns detected format, event count,
  preview list, and a suggested artist (term ID if a similar_text
  fuzzy match clears 85%, name otherwise). Logged-in users only.
- `data-machine-events/submit-artist-url` — re-probes server-side
  (never trusts client detection) and inserts a row in
  `pending_review` (or `scraping_failed`). Logged-in users only.
- `data-machine-events/approve-artist-url-submission` — admin only.
  Resolves the artist term (existing id, new name + wp_insert_term,
  or the submission's suggested term). Creates pipeline+flow via
  `datamachine/create-pipeline` + `datamachine/create-flow` mirroring
  the CityAbilities pattern, with universal_web_scraper handler.
  Triggers an immediate first run via `datamachine/run-flow`.
- `data-machine-events/reject-artist-url-submission` — admin only.
  Marks the row rejected with an optional reason.

SelectionMode alignment: the upsert step pins
`taxonomy_artist_selection = (string) $artist_term_id` (PRE_SELECTED)
and uses `SelectionMode::AI_DECIDES` constants by class reference for
venue/location/festival, `SelectionMode::SKIP` for promoter, category,
post_tag — no bare strings (#320 requirement).

Dedupe: SHA-256 over a normalized URL (lowercased scheme/host,
fragment stripped, trailing slash trimmed except root, default ports
removed). UNIQUE KEY on url_hash; duplicate preview/submit returns
`url_already_tracked`.

REST: `POST /wp-json/datamachine/v1/artist-url/{preview,submit,
{id}/approve,{id}/reject}`. Preview and submit reject direct-browser
navigations (no XHR / JSON Accept header → 404), in the spirit of

Admin moderation UI (`inc/Admin/ArtistUrlSubmissionsAdmin.php`): adds
"Artist URL Imports" under Events. Tabs filter by status (pending
review, approved, rejected, failed scrapes), inline forms for approve
(artist term id OR new name + schedule) and reject (reason). Forms
post to admin-post.php with nonces and delegate to the abilities.

Tests (`tests/Unit/ArtistUrlSubmissionsTableTest.php`,
`tests/Unit/ArtistUrlImportAbilitiesTest.php`): URL normalization,
url_hash dedupe, table CRUD, preview/submit rejection paths,
`artist_required` error from approve when no artist input is
provided, reject sets status + reason. Approve's pipeline-creation
integration is covered by curl verification against a live install
(matrix in the PR body) — recreating Data Machine core abilities in
isolated unit tests would be fake coverage.

The extrachill-events side (form change consuming preview + submit
abilities) ships separately once this merges.
@chubes4
chubes4 force-pushed the feat-320-artist-url-import branch from 96165f8 to 6f87b1f Compare May 27, 2026 02:23
@chubes4
chubes4 merged commit 64ce1c2 into main May 27, 2026
1 check passed
chubes4 added a commit to Extra-Chill/extrachill-events that referenced this pull request May 27, 2026
…EC-events side) (#118)

Adds a logged-in-user-only URL import field at the top of the
event-submission block. When the user pastes a tour page URL and tabs
away (or clicks 'Try URL' / hits Enter), the block calls the
`data-machine-events/preview-artist-url` REST ability and renders a
confirmation panel showing the events count, suggested artist, and the
first few extracted events. On confirm, the block calls
`data-machine-events/submit-artist-url` to queue the URL for admin
moderation.

Behavior:

- On preview success: manual form is hidden, confirmation panel
  appears with 'Found N events from [artist]. Submit for review?'.
- On 'url_already_tracked': clear message + reveal manual form so the
  user can submit a single-event instead.
- On 'no_events_found' or any other error: error message + reveal
  manual form.
- Cancel button restores the idle state and shows the manual form.
- 8s client-side timeout on probe; AbortController-based.
- Anonymous users never see the URL block (the DME ability rejects
  them anyway).

The existing manual single-event submission path is unchanged — when
the user doesn't touch the URL field, the form behaves byte-identically
to before. No admin-ajax, no jQuery; uses fetch + REST nonce.

REST endpoints + nonce are passed via data-* attributes on the
container so view.js stays static.

Depends on Extra-Chill/data-machine-events#322 (must be merged first
for the REST endpoints + abilities to exist).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant